Allow task_processing-launched pods to handle their own cleanup#218
Draft
Allow task_processing-launched pods to handle their own cleanup#218
Conversation
We have some usecases where we'd like a set of batches to run some cleanup code upon recieving a SIGTERM, so we'll need to remove our force-kill code and instead fallback to the default k8s behavior of giving Pods 30s after sending a SIGTERM before sending a SIGKILL to force-terminate :) I will also follow this up with a Tron documentation change that documents this behavior.
jfongatyelp
approved these changes
Jul 9, 2024
Contributor
jfongatyelp
left a comment
jfongatyelp
left a comment
There was a problem hiding this comment.
If we do this is it possible task_proc threads get blocked for 30s while just waiting for this deletion to finish? Or is python's async stuff smart enough to continue doing $other things in the meantime?
Member
Author
err, that's a good question - i think we'd probably want to make this an async request (which will require some testing since we don't currently use the client to make async requests anywhere) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We have some usecases where we'd like a set of batches to run some cleanup code upon recieving a SIGTERM, so we'll need to remove our force-kill code and instead fallback to the default k8s behavior of giving Pods 30s after sending a SIGTERM before sending a SIGKILL to force-terminate :)
I will also follow this up with a Tron documentation change that documents this behavior.